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ABSTRACT 



The views of rehabilitation consumers have not played a 
central role in shaping training programs for rehabilitation counselors. 
Likewise, partnerships between state vocational rehabilitation agencies and 
institutions of higher education have not been developed to their full 
potential. A planned collaboration between Montana Vocational Rehabilitation 
and Montana State University-Billings, so as to assess rehabilitation 
consumer satisfaction, is described in this paper. The primary purpose of the 
collaboration is to provide information of long-range value for 
rehabilitation service planning, and to improve the rehabilitation counselor 
training program at the collaborating school to make the program more 
relevant to the experience of rehabilitation consumers. Special emphasis is 
given to the program evaluation and curriculum development used in this 
program. Specific details on consumer satisfaction assessment procedures are 
outlined and the many benefits that assessment of consumer satisfaction 
offers to human services program evaluation are described. The role of the 
rehabilitation counselor in securing consumer feedback is discussed, since it 
is through the counselor that consumer needs are identified and served. Due 
to the link between service delivery and counselor preparation programs, 
models of collaborative consumer assessment surveys should be commonplace ; 
presently, such models are rare. (RJM) 
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Abstract 

Consumer feedback is a most important source of vocational rehabilitation outcome 
assessment data. However, the views of rehabilitation consumers have not been 
given a central role in shaping training programs for rehabilitation counselors. It is 
also true that partnerships between state vocational rehabilitation agencies and 
institutions of higher education have not been developed to their full potential. 

This paper describes a planned collaboration between Montana Vocational 
Rehabilitation and Montana State University-Billings to assess rehabilitation 
consumer satisfaction. The purposes of this collaboration are to provide 
information of long range value for rehabilitation service planning, and to improve 
the rehabilitation counselor training program at MSU-Billings by making it more 
relevant to the experience of rehabilitation consumers. 
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Rehabilitation Consumer Satisfaction Assessment: 

Collaboration Between a State Vocational Rehabilitation Agency 
and a Pre-Service Training Program 

There is strong public sentiment that government services should be made 
more responsive and accountable to consumers. The Rehabilitation Services 
Administration has encouraged state rehabilitation agencies and institutions of 
higher education to work together to make consumer outcomes assessment the 
basis of program planning. In a recently published Region VIII Education/Training 
Collaboration Plan the importance of collaboration between educational programs 
and service delivery systems, and the need to connect rehabilitation counselor 
training with employment outcomes for consumers, were identified as high priority 
goals (Region VIII RCEP, 1996). Unfortunately, at the present time it is difficult to 
locate models of assessment collaboration. 

Research into the effectiveness of competency based learning in rehabilitation 
counseling has supported the use of a diversified pre-service curriculum (Harrison & 
Lee, 1979; McGowan & Porter, 1967; Muthard & Salomone, 1969; Scorzelli, 1975). 
Rehabilitation counselor education programs draw curricula from the fields of law, 
medicine, psychology, philosophy, and history. It is also necessary for rehabilitation 
counselors in training to complete comprehensive field-based learning experiences. 
As part of their preparation, graduate students in rehabilitation counseling must 
become proficient in numerous specific areas of knowledge and skill. 

This paper describes a program evaluation and curriculum development 
approach that is being used by Montana Vocational Rehabilitation and Montana 
State University-Billings to promote curriculum development in a Master of Science 
in Rehabilitation Counseling program. The curriculum development approach 
described in this paper came about as a result of a Rehabilitation Services 
Administration Region VIII meeting in Denver, Colorado, November 6-7, 1996 
(Region VIII RCEP, 1996). 

Curriculum Development - 
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As in all disciplines, curriculum development is an important challenge for 
rehabilitation educators. Professional rehabilitation counselors serve a multifaceted 
role, utilizing diverse skills in several functional domains (Emener & Rubin, 1980; 
Harrison & Lee, 1979; Matkin, 1983). CORE specifies 51 specific competencies as 
required for Masters of Rehabilitation Counseling programs (Council on 
Rehabilitation Education, 1996). Educators must select courses that address the 
necessary competency areas and present them to students in a manner that reflects 
the contemporary challenges of the rehabilitation counselor's professional role. 

Crystal (1981) called attention to the importance of the interrelationships 
among components of the rehabilitation process and proposed an integrative model 
for training and practice. Crystal stated, "By understanding the way variables in the 
rehabilitation process interact, counselors can develop skills which enable them to 
anticipate potential problems and deal with them before they interfere with the 
client's rehabilitation program" (p. 194). Recommended components of an integrated 
curriculum were (a) assessing client needs, (b) developing counseling competencies, 
(c) understanding the rehabilitation process, (d) understanding the environment, (e) 
developing caseload management skills, and (f) developing self-monitoring and 
evaluation skills. 

The Master of Science in Rehabilitation Counseling Program at Montana State 
University - Billings has been organized into five learning tracks and four sequential 
levels. The learning tracks and objectives have been defined as follows: (a) World of 
Work - To provide students with awareness of the significance of vocational activity 
in society and individual experience, and to promote understanding of the 
relationship of vocational/career development and choice to the broader contexts of 
human personality, community, society, and culture; (b) Professional Counselor - To 
provide students with knowledge of major counseling theories and their 
relationship to process and technique. To instill awareness of the ethical dimension 
inherent in all counseling work. To provide students with strategies and skills 
essential for the initiation, maintenance, and termination of counseling 
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relationships; (c) A gency Life - To provide students with knowledge of the 
historical/legal foundations of rehabilitation service delivery systems, both public 
and private. To provide students with the necessary knowledge, skills, and attitudes 
to function successfully in a professional rehabilitation counseling environment; (d) 
Living With Disability - To inform students concerning the medical and 
psychological dimensions of severely disabling conditions. To provide an 
understanding of the impact of severe disability on individual development, social 
relations, and family life; and (e) Assessment - To provide students with knowledge 
of the traditions and methods of systematic empirical reasoning, in both classroom 
and applied settings. 

The four sequential levels of the curriculum are: (a) Introduction - To provide 
students with an orientation to the basic knowledge, attitudes, and discourse 
conventions of Rehabilitation Counseling. To provide a fundamental overview of 
the lives and challenges of consumers and the roles and functions of Rehabilitation 
Counselors; (b) Specialization - To provide students with advanced skills and 
knowledge related to the specialized problems, settings, and situations encountered 
in professional Rehabilitation Counseling work; (c) Community - To provide 
students with knowledge and practical experience related to the role of rehabilitation 
counselors in the community and the lives of consumers within the community; 
and (d) Integration - To provide students with a full integrative sense of the 
relationship between the fundamental and advanced skills of Rehabilitation 
Counseling. To provide students with an understanding of the relationship between 
the constituent knowledge areas of Rehabilitation Counseling. The learning tracks 
and levels of the curriculum design appear in Figure 1. 

It is important that training programs evolve in pace with the changing 
demands of the rehabilitation counseling role. Emener and McFarlane (1985) have 
pointed out that changes in society such as technological advancement and 
economic, political, and cultural developments present new informational contexts 
for rehabilitation educators to consider. Curriculum infusion and creation of new 




s 



Consumer Satisfaction 

6 

courses are two approaches educators have used to integrate new information into 
educational programs (Emener, Rollins, Fischer & Rubin, 1993; Rubin, Pusch, 
Fogarty, & McGinn, 1995; Schaller & DeLaGarza, 1994). 

Educators must consider information from a variety of sources when 
determining what new information should be integrated into a rehabilitation 
program. CORE periodically reviews accredited programs and makes specific 
recommendations for improvement. Faculty must remain aware of 
recommendations from the Rehabilitation Services Administration which pertain to 
programs that administer federal training grants. Credentialing bodies such as the 
Commission for Rehabilitation Counselor Certification (CRC) also provide input 
concerning content of pre-service curriculum. Additionally, faculty must be guided 
by the research findings and opinions of peer groups such as the National Council of 
Rehabilitation Educators (NCRE). To these established sources of feedback and 
recommendations we believe consumer generated information must be added. 

Many programs access consumer information by means of a program advisory 
committee. While advisory committees serve an invaluable purpose, it is our 
position that an empirical investigation of consumer opinion would add much to 
the program development picture. 

This consumer satisfaction assessment will need to be matched to information 
relative to consumer expectations or needs as they emerge as well as the readiness of 
the agency staff to provide appropriate services. We are living in an era that brings 
new and complex challenges. It is important for the pre-service program to identify 
the issues related to the needs of the people who will benefit from the rehabilitation 
professional's readiness and expertise. There is also the need to ascertain the needs of 
the service delivery personnel. Preparation at the pre-service arena centers on the 
levels of knowledge and skills needed to deliver effective services within multiple 
rehabilitation environments. One avenue of information is the public agency 
personnel, who serve as gatekeepers of information about the needs of the 
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consumers who apply for services as well as the relevance of specific skills within 
their own profession. 

Consumer Satisfaction Assessment - 

Authors have pointed out that assessment of consumer satisfaction is a 
necessary component of human services program evaluation. Beyond the obvious 
fact that federal legislation requires state rehabilitation agencies to conduct 
consumer satisfaction surveys, other compelling reasons exist. Without consumer 
feedback, service providers would have only the perspective of their own 
observations to consider when determining the usefulness of programs. No matter 
how hard providers might try to be objective, the possibility of bias remains. 

Another important reason for assessing consumer satisfaction is the fact that in 
public service the standard market forces cannot dictate improvement. Unlike 
customers in the private sector, consumers of rehabilitation services ordinarily have 
only one provider available. Thus, for rehabilitation providers to achieve high 
levels of quality it is necessary to conduct detailed outcomes assessment, particularly 
consumer satisfaction. Additionally, it is important to remember that rehabilitation 
services exist in response to a public consensus of need, as stated in law. Beyond the 
quantitative and somewhat narrow performance measures mandated in law, 
consumer satisfaction assessment offers the only practical opportunity to determine 
the extent to which the law is accomplishing its original purposes. 

Rehabilitation counselors are central figures in the state-federal program of 
vocational rehabilitation. It is through the counselor that consumer needs are 
identified and served, and it is in the context of a relationship with the counselor 
that all consumer interactions with the service delivery system take place. Such is 
the importance of the counselor to the rehabilitation process that legislation 
mandates that state agencies employ only qualified rehabilitation counselors. The 
task of training qualified rehabilitation counselors is met by accredited rehabilitation 
education programs. Because of the obvious link between service delivery and 
educational preparation programs, it would be reasonable to expect that models of 
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collaborative consumer assessment surveys would exist. Unfortunately, such 
models are uncommon and difficult to find. 

The approach described in this paper proposes to utilize the Client 
Satisfaction Questionnaire (CSQ-8) developed by Larsen, Attkisson, Hargreaves, and 
Nguyen (1979). The CSQ-8 is an eight item form that has been identified as having 
potential for use in rehabilitation settings (Koch & Merz, 1995). The CSQ-8 has been 
reported with internal consistency coefficients ranging from .83 to above .90 
(Attkisson & Zwick, 1982; Greenfield, 1983). Developed originally for use in 
community mental health centers, the CSQ-8 has been shown to correlate with 
other outcome measures, such as numbers of sessions attended and therapist ratings 
of improvement (Attkisson & Zwick, 1982; Wilkin, Hallam, & Doggett, 1992). The 
CSQ-8 offers strong advantages as a consumer assessment tool for collaborative use 
by service delivery and educational preparation programs; it consists of eight items 
that contribute to a global assessment score, it solicits qualitative information in the 
form of comments, and situation specific items may be added to the original eight 
(Daly & Flynn, 1985; Greenfield, 1983). For the purpose of obtaining consumer 
comments and suggestions for improvement of the rehabilitation counseling 
program at MSU-Billings, five items relating specifically to curriculum learning 
tracks were added to the CSQ-8, and appear in Figure 2. 

One persistent problem associated with consumer satisfaction surveys is that 
data tends to be highly positive with negatively skewed distributions. Authors are 
uncertain of the causes of this phenomenon. It has been suggested that most 
consumers feel they are supposed to give "grateful testimonials", while a smaller 
percentage reflect a rigid tendency to criticize (Larsen, Attkisson, Hargreaves, and 
Nguyen, 1979; Peterson & Wilson, 1992). Several approaches are available for 
reducing bias and obtaining useful information. Surveying a cross-section of 
consumers who represent all levels of the rehabilitation process and extending the 
data collection period may reduce the likelihood of missing consumers who miss 
appointments or drop out of the program. It is also important to specifically request 
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consumers to provide qualitative suggestions for program improvement. In that 
way consumers are oriented toward providing useful suggestions, regardless of 
whether their opinion is predominantly positive or negative. 

The benefits of a collaborative assessment of consumer satisfaction are many. 
We anticipate the improvement of the MSU-Billings graduate program in 
rehabilitation counseling on the basis of consumer feedback. Counselors in training 
need to know and be influenced by the views of rehabilitation consumers, not 
simply those who claim to represent consumers. The coursework that comprises 
their preparation should focus on issues that are most important to consumers. 

Quality control benefits for Montana Vocational Rehabilitation will come 
from statistical assessment of consumer views. Longitudinal assessment of the 
impact of new management and service delivery models will be made possible by 
the development of norms. Most consumer satisfaction information is difficult to 
interpret because of the combined problem of negatively skewed distributions and 
the absence of norms (Lehman & Zastowny, 1983). It is not very informative to learn 
that a certain percentage of consumers report satisfaction with services without a 
normative basis of comparison. Through development of local norms and 
transformation of data it will be possible to make meaningful inferences regarding 
the impact of program initiatives (Peterson & Wilson, 1992). 

Rehabilitation Counselors as Gatekeepers of Information 

Assessment for, and determination of, eligibility for services and programs 
and feasibility of vocational or independent living objectives are a part of the job of 
the rehabilitation counselor. This requires understanding the medical aspects of 
disabilities, functional capacities and limitations related to disability, and appropriate 
treatments or interventions. The psychosocial aspects of disabilities is integral to the 
rehabilitation process, including the impact of disability on the individual and family 
(CORE, 1996). 

During the process of application and determination of eligibility, the 
rehabilitation counselor gathers information about the needs of the consumers. 
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Some of these needs are not met because of lack of resources or the requests fall 
outside of the legislative mandates for the agency. This information may be vital to 
the disability community, who must advocate for resources and legislation within 
their state and locality. At present, it is not systematically gathered and captured for 
reporting purposes. We propose to work with the agency and their advisory groups 
to determine how to approach gathering and analyzing the materials. While 
maintaining consumer confidentiality, the data gathering done at intake can be 
analyzed by researchers and reported as concrete evidence of the needs of consumers 
throughout the state of Montana. 

Rehabilitation Skills Inventory 

The Council on Rehabilitation Education (CORE) identified key competencies 
for rehabilitation counselors (1996). These competencies were based on assumptions 
of job responsibilities within the profession. The results of a national survey of state 
job descriptions rated the following skill areas a high level of importance: vocational 
services, case management, medical and psychosocial issues, individual counseling, 
and assessment (Allen, Turpin, Garske, & Warren-Marlatt, 1996). 

Rehabilitation counselors are required to possess a "breadth and depth of 
technical competencies and to exercise great and continuing effort toward maximal 
case outcomes" (Wright, Leahy, & Shapson, 1987, p. 108). For the rehabilitation 
counselor, these are in constant flux as special populations emerge, research leads to 
innovative programs, and legislative mandates change with the social and political 
climate (Leahy, Szymanski, & Linkowski, 1993). Counseling is the basic function 
with many other responsibilities in the job description. This description of duties for 
any individual counselor will vary according to client needs and resources. The 
rehabilitation needs of persons with disabilities are multiple and complex and they 
can be denied quality services if counselors do not possess adequate commitment, 
knowledge and skills to meet these needs (Shapson, Wright, & Leahy, 1987). 

An area that defines rehabilitation counselors' competence is mastery of 
specific job-related skills. Counselors meet these expectations through the 
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development of special expertise and continuing professional education (Blocker, 
1987). For the rehabilitation counselor, these are in constant flux as special 
populations emerge, research leads to innovative programs, and legislative 
mandates change with the social and political climate (Leahy, et. al). 

The key competencies identified by CORE were based on assumptions of job 
responsibilities within the profession. The rehabilitation counselor in the public 
arena functions as both counselor and coordinator. They are professionals who serve 
people with disabilities through delivery of multifaceted and integrated services 
(CORE, 1996). The results of a national survey of state job descriptions rated the 
following skill areas a high level of importance: vocational services, case 
management, medical and psychosocial issues, individual counseling, and 
assessment (Allen, et. al, 1996). 

Education and years of experience have been reported to be predictors of better 
job performance and improved client outcomes in the rehabilitation service delivery 
process (Cook & Bolton, 1992; Szymanski & Parker, 1989). This is complicated by the 
fact that work settings define essential competencies and the job descriptions vary 
according the needs of clients and the resources in the community and agency 
(Danek, Wright, Leahy, & Shapson, 1987; Wright, et. al, 1987). 

Determining what competencies are important is one part of this inventory, 
the second is measuring attainment of these competencies. The Rehabilitation Skills 
Inventory (RSI) is one tool developed to measure these competencies among 
rehabilitation counselors. The RSI was constructed to represent the domains of 
professional competencies needed for rehabilitation practice. It collects data 
regarding these competencies by assessing the persons' perceived attainment of them 
as well as their perception of the importance of the particular skill. This addresses 
not only the stated skills that were outlined by CORE but also allows the varying 
nature of job responsibilities among rehabilitation professionals. Importance means 
the extent to which a professional skill is relevant to the primary work role and how 
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critical the use of this skill is to clients' rehabilitation. Importance in this instance 
refers to the criticalness of the competency (Wright, et. al). 

According to Wright, et al., this type of tool can have numerous purposes, 
including: use by in-service trainers for planning; review by University educators for 
the competency basis of curriculum design and course content; and justification of 
the need for qualified rehabilitation professionals by the community. It closes the 
loop started by the identification of consumer needs with information that accounts 
for the system's ability to respond to those needs. 

Summary 

Collaboration between Montana Vocational Rehabilitation and Montana State 
University-Billings to assess consumer satisfaction would be in the best interests of 
rehabilitation consumers. Assessment of consumer satisfaction will help build a 
stronger pre-service curriculum for the preparation of rehabilitation counselors. 
Long-range assessment will result in the development of norms, thus making it 
possible for Montana Vocational Rehabilitation to assess the impact of new 
management and service delivery models. 
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Figure 2. Consumer Satisfaction Questionnaire 

Please help us improve our program by answering some questions about the services you have 
received from Montana Vocational Rehabilitation. We are interested in your honest opinions, whether 
they are positive or negative. Please answer all of the questions , We also welcome your comments and 
suggestions for improvement of the rehabilitation program. Thank you, we appreciate your help. 
CIRCLE YOUR ANSWER: 



1 . 



How would you rate the quality of service you received? 



4 

Excellent 



3 

Good 



2 

Fair 



1 

Poor 



2. Did you get the kind of service you wanted? 
1 2 
No, definitely not No, not really 



3 

Yes, generally 



4 

Yes definitely 



3. To what extent has our program met you needs? 
4 3 

Almost all of my Most of my needs 

needs have been have been met 

met 



2 

Only a few of my 
needs have been 
met 



1 

None of my needs 
have been met 



4. If a friend were in need of similar help, would you recommend our program to him/her? 

12 3 4 

No, definitely not No, I don't think so Yes, I think so Yes, definitely 



5. How satisfied are you with the amount of help you received? 

12 3 4 

Quite Indifferent or Mostly satisfied Very satisfied 

dissatisfied mildly dissatisfied 

6. Have the services you received helped you to deal more effectively with your problems? 

4 3 2 1 

Yes, they helped Yes, they helped No, they really No, they seemed to 

a great deal somewhat didn't help make things worse 



7. In an overall, general sense, how satisfied are you with the service you received? 

4 3 2 1 

Very Mostly Indifferent Quite 

satisfied satisfied mildly dissatisfied dissatisfied 



8. If you were to seek help again, would you come back to our program? 
1 2 3 

No, definitely not No, I don't think so Yes, I think so 



4 

Yes definitely 



COMMENTS & SUGGESTIONS FOR IMPROVEMENT: 
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Please help us improve counselor preparation by answering the questions below. We are 
interested in your honest opinions, whether they are positive or negative. Please answer all of the 
questions. In order to completely understand your views it is important that you include comments and 
suggestions for improvement of counselor preparation. Thank you very much. 

1. 1 feel my career and job placement needs have been well served. 

12 3 4 

No, definitely not No, not really Yes, generally Yes definitely 

COMMENTS & SUGGESTIONS FOR IMPROVEMENT: 



2. My relationship with my rehabilitation counselor(s) has been helpful and productive. 

4 3 2 1 

Yes, it helped Yes, somewhat No, not really No, it made 

a great deal things worse 

COMMENTS & SUGGESTIONS FOR IMPROVEMENT: 



3. Information and services have been provided to me in a timely, sufficient, and courteous manner. 
12 3 4 

No, definitely not No, not really Yes, generally Yes definitely 

COMMENTS & SUGGESTIONS FOR IMPROVEMENT: 



4. 1 feel my counselor(s) understands my disability and what it means in my life. 

4 3 2 1 

Yes, very much Yes, somewhat Not very much No, not at all 

COMMENTS & SUGGESTIONS FOR IMPROVEMENT: 



5. 1 feel the decisions concerning my case were reasonable and based on good information. 

12 3 4 

No, definitely not No, not really Yes, generally Yes definitely 

COMMENTS & SUGGESTIONS FOR IMPROVEMENT: 
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